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Abstract 

This  study  tested  the  hypothesis  that  defense  cost  variances  reported  on  the  Cost 
Performance  Report  are  normally  distributed.  The  DOD  requires  that  all  defense  cost 
variances  which  breech  a  pre-specified  threshold  be  investigated.  The  present  variance 
investigation  model  has  been  criticized  because  it  can  prompt  frivolous  investigations.  In 
theory,  statistical  models  could  reduce  the  number  of  frivolous  investigations,  but  they  are 
not  used  because  they  require  too  much  information  about  the  cost  variance,  including  its 
distributional  form.  Often  such  models  assume  a  normal  distribution,  but  researchers  have 
shown  that  the  models  do  not  work  properly  if  the  assumption  is  fallacious.  Two  prior 
studies  have  investigated  the  normality  of  cost  variances  with  mixed  results,  and  neither 
investigated  defense  cost  variances.  Here,  fifty  series  of  cost  variances  from  two  defense 
contracts  were  extracted  from  Cost  Performance  Reports  and  evaluated  using  four 
popular  tests  of  normality  (Bowman-Shenton,  Shapiro-Wilks,  Kolmogorov-Smimov,  and 
Chi-square).  The  results  show  that  the  vast  majority  of  the  series  of  cost  variances  were 
not  normally  distributed.  These  results  were  insensitive  to  the  normality  test  used  and  to 
the  effects  of  inflation.  The  statistical  variance  investigation  models  may  still  be  used,  but 
normality  should  not  be  assumed. 
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SOME  EMPIRICAL  EVIDENCE  ON  THE  NON-NORMALITY 


OF  COST  VARIANCES  ON  DEFENSE  CONTRACTS 


I.  Introduction 


The  Issue 

Since  the  end  of  the  Cold  War,  pressure  for  improved  cost  efficiency  on  defense 
projects  has  been  enormous.  The  Department  of  Defense  (DOD)  has  responded  by 
reducing  its  forces  and  promoting  policies  such  as  “fee-for-service”  to  make  defense 
organizations  more  cost  efficient.  The  DOD  has  also  examined  its  defense  acquisition 
procedures  in  order  to  streamline  the  acquisition  process  and  take  advantage  of  innovative 
and  cost-efficient  practices  in  industry,  such  as  just-in-time  production  and  activity-based 
costing. 

One  area  that  has  tremendous  potential  for  improved  cost  efficiencies  is  the  control  of 
cost  growth  and  the  elimination  of  cost  overruns  on  defense  projects.  For  example,  based 
on  a  review  of  over  one  hundred  major  weapon  systems  since  the  mid  1960s,  Drezner  et 
al.  (10:xiii)  report  that  the  average  cost  growth  has  fluctuated  around  20  percent. 

Similarly,  based  on  an  analysis  of  hundreds  of  defense  contracts  since  the  1960s, 
Christensen  (3:30)  reports  that  the  average  cost  overrun  on  defense  contracts  is  about  18 
percent.  These  findings  are  particularly  disappointing  because  these  cost  problems  have 
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continued  despite  numerous  acquisition  initiatives  and  policies  since  the  1960s  designed  to 
control  them  (Drezner  et  al.  1993,  10:29). 

One  of  these  policies  requires  that  defense  contractors  comply  with  DOD 
Cost/Schedule  Control  Systems  Criteria  (C/SCSC),  also  known  as  “the  criteria.”  Simply 
put,  the  criteria  are  internal  controls  which  require  the  development  and  use  of 
performance  budgets  to  manage  a  defense  project.  Although  the  criteria  have  been  widely 
supported  as  a  sound  project  management  tool,  they  have  been  over-implemented  by  the 
military  services  and  are  now  being  revised  to  reduce  the  administrative  cost  that  over¬ 
implementation  has  created. 

One  area  addressed  by  the  criteria  that  has  been  over-implemented  involves  an 
excessive  analysis  of  cost  variances.  The  criteria  require  the  defense  contractor  to  analyze 
“significant  variances,”  and  specify  that  a  significant  variance  is  one  that  breeches  a  pre¬ 
determined  threshold,  expressed  as  either  a  percentage,  a  dollar  amount,  or  as  a 
combination  of  the  two.  (8:3-17).  For  example,  a  cost  variance  may  be  defined  as 
significant  if  it  exceeds  10  percent  of  the  budget,  or  exceeds  $10,000,  or  exceeds  10%  and 
$10,000.  When  a  breech  occurs,  the  contractor  is  required  to  investigate,  report  the 
cause,  and  implement  a  corrective  action  plan  if  possible.  Although  the  use  of  simple 

thresholds  to  determine  when  to  investigate  a  cost  variance  is  simple,  it  can  become  an 

« 

administrative  burden  when  the  threshold  is  applied  mechanically  to  all  levels  of  work  on 
the  contract.  Unfortunately,  this  has  been  the  experience  on  defense  contracts,  and 
contractors  have  sought  relief  from  such  requirements  for  many  years. 
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The  academic  literature  describes  several  statistical  cost  variance  investigation  models 
which  are  reported  to  be  superior  to  the  simple  variance  investigation  model  described 
above  (Kaplan,  1975, 20).  Assuming  a  knowledge  of  the  distributional  properties  of  a 
cost  variance,  the  statistical  models  use  probability  theory  to  signal  an  investigation  only 
when  the  marginal  benefit  of  correcting  the  problem  exceeds  the  marginal  cost  of  the 
investigation.  Thus,  the  use  of  these  models  on  defense  contracts  has  the  potential  to 
reduce  the  number  of  firivolous  defense  cost  variance  investigations  that  the  simple  cost 
variance  investigation  model  now  requires.  However,  based  on  a  review  of  defense 
contracts  managed  by  the  Air  Force,  Hoang  and  Quick  (1993,  15;vii)  report  that  the 
statistical  models  are  rarely  used.  This  finding  is  consistent  with  reports  that  statistical 
cost  variance  investigation  models  are  rarely  used  in  the  civilian  sector  (Koehler,  1968, 
22). 

The  Research  Problem 

One  reason  suggested  for  not  using  the  statistical  models  involves  the  requirement  that 
the  distributional  properties  of  the  cost  variance  be  known  in  advance  (Boer,  1984,  1; 
Gribbin  and  Lau,  1991,  13).  For  example,  these  models  are  often  described  using  the 
assumption  that  the  cost  variance  is  normally  distributed.  However,  Gribbin  has  recently 
shown  that  if  this  assumption  is  erroneous,  then  the  variance  investigation  signal  can  be 
suboptimal  (Gribbin,  1989,  12). 
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In  a  recent  study  of  cost  variances  at  a  medium  size  manufacturing  plant,  Gribbin  and 
Lau  report  that  cost  variances  experienced  there  were  not  always  normal  (Gribbin  and 
Lau,  1991,  13).  The  only  other  study  of  the  distributional  properties  of  cost  variances 
reported  similar  results  (Jacobs  and  Lorek,  1980,  17).  Thus,  Gribbin  and  Lau  caution 
current  and  potential  users  of  the  statistical  cost  variance  investigation  models  that  the 
assumption  of  normality  is  not  always  appropriate,  and  recommend  that  their  research  be 
replicated  in  other  settings. 

Given  the  increasing  importance  of  cost  efiBciency  in  defense,  and  the  widely 
recognized  problem  with  the  current  defense  cost  variance  investigation  model,  this  study 
replicates  the  research  of  Gribbin  and  Lau  using  data  from  completed  and  on-going 
defense  contracts.  Specifically,  it  investigates  the  normality  of  cost  variances  reported  on 
two  defense  contracts. 


Hypothesis  Statement 

An  appropriate  hypothesis  in  null  form  is: 

Ho:  Defense  cost  variances  are  normaUy  distributed. 

If  the  hypothesis  is  supported,  the  use  of  statistical  cost  variance  investigation  models 
which  require  the  assumption  of  normally  distributed  cost  variances  should  be  encouraged 
on  defense  contracts.  If  not  supported,  then  the  statistical  models  may  still  be  beneficial, 
but  only  with  non-normal  distributions  that  more  closely  fit  defense  cost  variances. 
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Conclusion 


The  increased  emphasis  on  cost  eflBciency  in  defense,  and  the  wide-spread 
dissatisfaction  with  the  present  variance  investigation  model  used  on  defense  contracts, 
have  prompted  this  study.  Further,  based  on  their  analysis  of  nondefense  cost  variances, 
Gribbin  and  Lau  (13)  conclude  that  the  indiscriminate  use  of  the  normality  assumption  in 
statistical  cost  variance  investigation  models  is  inappropriate,  and  recommend  more 
empirical  research  into  the  distributional  properties  of  cost  variances.  This  study 
replicates  Gribbin  and  Lau’s  (13)  study  using  defense  cost  variances. 

The  remaining  chapters  review  the  relevant  literature  (Chapter  H),  describe  the 
methodology  (Chapter  El),  report  the  results  of  the  hypothesis  test  (Chapter  IV),  and 
summarize  the  project  and  its  implications  (Chapter  V). 
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n.  Literature  Review 


Introduction 

As  indicated  in  Chapter  I,  statistical  cost  variance  investigation  models  are  considered 
superior  to  the  present  model  commonly  used  on  defense  contracts.  Because  the 
statistical  models  are  based  on  probability  theory  and  compare  marginal  benefits  to 
marginal  costs  before  prompting  an  investigation,  the  use  of  these  models  would  likely 
reduce  both  the  number  of  fiivolous  variance  investigations  and  the  cost  of  managing  a 
defense  contract. 

However,  the  statistical  models  often  assume  that  cost  variances  are  distributed 

normally  (1:48,  51;  18:24;  23:140;  25:66-78;  26:728),  which  may  not  be  the  case.  If 

the  cost  variances  are  not  normal,  then  an  investigation  signal  fi-om  a  statistical  model 

which  assumes  normality  may  still  prompt  a  fiivolous  investigation: 

Gribbin  has  shown  recently  that  if  the  cost  variances  are  indeed  non-normal,  then 
assuming  normality  instead  of  modeling  the  non-normality  correctly  can  lead  to 
significantly  inferior  cost  variance  investigation  decisions.  (Gribbin  and  Lau, 
1991,  13:88) 

Thus,  this  study  tests  the  null  hypothesis  that  cost  variances  reported  on  defense 
contracts  are  normally  distributed.  In  this  chapter,  the  relevant  academic  literature  which 
proposes  various  statistical  cost  variance  models  is  summarized.  Although  these  models 
appear  to  be  improvements  over  the  simple  model,  surveys  indicate  that  they  are  rarely 
used  in  industry  (Laudeman  and  Schaeberle,  1983,  24;  Gaumnitz  and  Kollaritsch,  1988 
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1 1).  Therefore,  this  chapter  also  reviews  various  reasons  given  for  not  using  the  models. 
One  of  these,  of  course,  is  the  possible  fallacious  assumption  of  normality.  The  final 
section  of  this  chapter  reviews  the  only  two  published  studies  which  have  tested  the 
normality  assumption. 


Cost  Variance  Investigation  Decision  Models 

The  academic  literature  describes  several  statistical  cost  variance  investigation  models 
which  are  reported  to  be  superior  to  the  simple  variance  investigation  model.  Kaplan 
(20;3 1 1-337)  surveyed  the  accounting,  statistics,  and  management  science  literature 
dealing  with  these  models,  and  developed  a  taxonomy  that  organizes  the  models  along 
dimensions  which  form  the  following  table. 


Table  1.  A  Taxonomy  of  Variance  Investigation  Models  (Kaplan,  1975, 20) 


Costs  and  Benefits  of  Investigation  Not 
Considered 

Costs  and  Benefits  of  Investigation 
Considered 

Single-Period 

2^annetos  (1964),  Juers  (1967) 

Koehler  (1968),  Luh  (1968),  Probst  (1971), 
Buzby(1974) 

Duncan  (1956) 

Bierman,  Fouraker,  and  Jaedicke  (1961) 

Multi-Period 

Cumulative-Sum  Chart  as  in  Page  (1954) 
Also  Barnard  (1959),  Chemoff  and  2^cl^ 
(1964) 

Duvall  (1967),  Kaplan  (1969) 

Dycknm  (1969),  Bather  (1963) 

One  dimension  classifies  the  models  by  the  number  of  observations  they  require.  The 
other  dimension  classifies  the  models  by  whether  or  not  the  costs  and  benefits  of  the 
investigation  are  considered.  Thus,  the  table  places  variance  investigation  models  into 
four  categories,  where  each  category  includes  examples  of  variance  investigation  models 
proposed  by  researchers.  Because  Kaplan  (1975,  20)  describes  these  examples  in  detail. 
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they  will  not  be  repeated.  A  brief  description  of  these  categories  and  their  relationship  to 
the  normality  assumption  follows. 

Single-period  Models  with  No  Cost-benefit  Comparison.  This  type  of  model  is  the 
most  common,  where  current  cost  variances  which  breech  a  pre-determined  threshold  are 
investigated.  In  some  cases,  a  control  chart  approach  is  used,  where  the  cost  variance  is 
assumed  to  be  a  random  variable  with  a  normal  probability  distribution,  and  the  threshold 
is  defined  as  a  set  number  of  standard  deviations  fi’om  the  expected  value  of  the  cost 
variance. 

On  defense  contracts,  thresholds  are  usually  formally  specified  as  a  simple  percentage, 
a  dollar  amount,  or  both  (8;3-17)  on  the  Contractor  Data  Requirements  Listing  (CDRL). 
In  addition,  thresholds  can  be  revised  by  contractor  and  government  management  during 
the  life  of  the  contract.  Hoang  and  Quick  (15)  report  that  modeling  the  cost  variance  as  a 
random  variable  is  almost  never  done  (15:57),  and  in  some  cases  thresholds  are  simply 
copied  from  the  CDRLs  of  prior  contracts  (15:62). 

Multi-period  Models  with  No  Cost-benefit  Comparison.  One  way  to  improve  the 
single-period  model  is  to  include  previous  observations.  The  expectation  is  that  by 
examining  the  trend  of  variances,  a  significant  problem  may  be  detected  sooner,  especiaUy 
when  no  individual  variance  by  itself  may  exceed  a  threshold.  Kaplan  (19:151-153) 
reports  that  the  “cumulative  sum  procedure”  is  the  most  common  model  of  this  kind, 
where  variances  are  often  assumed  to  be  normally  distributed  (19: 15 1-153).  Furthermore, 
defense  policy  does  not  prevent  the  use  of  this  type  of  model,  but  Hoang  and  Quick  (15) 
report  that  its  use  is  rare. 
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Models  with  Cost-benefit  Comparisons.  Regardless  of  the  periods  included,  signaling 
an  investigation  only  when  the  expected  benefit  exceeds  the  expected  cost  is  an 
improvement  over  the  basic  model,  because  the  control  chart  approach  does  not  formally 
include  costs  and  benefits.  Clearly,  these  models  require  a  lot  of  information,  including 
estimates  of  the  cost  of  the  investigation,  the  benefit  of  correcting  an  out-of-control 
process,  the  cost  of  correcting  the  out-of-control  process,  and  the  probability  that  the 
process  is  out-of-control.  In  addition,  the  assumption  that  the  cost  variance  is  normally 
distributed  is  commonly  made  in  the  literature  which  describes  this  class  of  models  (e.g., 
Kaplan,  1982,  19:337-338). 

Assessment.  Each  of  these  categories  of  models  has  their  strengths  and  weaknesses. 
The  basic  model,  which  is  used  on  defense  contracts,  is  the  easiest  to  implement  and 
requires  much  less  information  than  the  other  models.  However,  if  the  information  is 
available,  the  models  which  include  multiple  periods  and  a  cost-benefit  comparison  are 
clearly  superior  by  reducing  the  amount  and  cost  of  fiivolous  investigations.  The  main 
problem  with  the  more  elaborate  models  is  the  additional  information  required  to  use 
them.  But  the  defense  policy  which  requires  a  cost  variance  investigation  does  not 
prohibit  the  use  of  the  more  elaborate  models. 

Normality  Studies 

As  indicated  in  the  preceding  section,  the  assumption  of  normality  is  fi-equently 
included  in  descriptions  of  the  statistical  cost  variance  investigation  models.  After 
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completing  his  survey  of  the  cost  variance  investigation  model  literature,  Kaplan  (20) 
concludes  that 

The  final judgment  on  the  appropriateness  of formal  statistical  and  mathematical 
models  for  cost  variance  analysis  must  be  based  on  empirical  studies.  To  date, 
little  such  evidence  is  available.  (20: 148) 


The  validity  of  the  normality  assumption  is  an  empirical  question.  As  indicated  in  Table  2, 
only  two  reported  studies  have  explored  this  question.  Each  of  these  will  now  be 
described. 


Table  2.  Cost  Variance  Normality  Studies 


Researchers 

(Year) 

Variances  (amount) 

Normality  tests  used 

Results  at  a=^  .05 

Jacobs  &  Lorek 
(1980) 

Material  and  utilities  usage 
(1 1  daily ,  9  weekly)  from 
a  grain  processing  firm 

Skewness,  Kurtosis, 
Kolmogorov-Smimov 

None  of  the  daily  and  7  of 

9  weekly  variances  tested 
normal. 

Gribbin  &  Lau 
(1991) 

Direct  labor  efBciency  in 
dollars  and  percent  (32  to 

43  months  in  each  of  14 
production  departments) 

Bowman-Shenton 
Shapiro- Wilk 

7  of  14  of  the  dollar  and  1 
of  the  14  percentage 
variances  tested  normal. 

Jacobs  and  Loreck.  Jacobs  and  Loreck  (17)  were  the  first  to  investigate  the  normality 
of  cost  variances.  In  their  study  of  usage  variances  experienced  on  several  processes  at  a 
grain  processing  firm,  1 1  series  of  daily  variances  and  9  series  of  weekly  variances  were 
tested  for  normality  using  the  Kolmogorov-Smimov  test  and  moment  tests  (skewness  and 
kurtosis).  These  tests  and  other  normality  tests  will  be  described  in  Chapter  m.  A  usage 
variance  is  the  difference  between  a  budgeted  and  actual  quantity  used  in  a  process. 
Usually  this  difference  is  multiplied  by  the  standard  price  per  unit.  In  this  case,  the  authors 
reported  that  price  data  were  not  available  to  them.  Also,  it  is  not  clear  how  many 
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variances  were  included  in  a  series  and  if  the  samples  were  random.  Given  these 
limitations,  the  normality  hypothesis  was  rejected  for  all  of  the  daily  variances,  and 
accepted  for  7  of  the  9  weekly  variances  at  the  .05  significance  level.  Thus,  the  authors 
concluded  that  usage  variances  may  not  always  be  normally  distributed. 

Gribbin  and  Lau.  Gribbin  and  Lau  (13)  investigated  the  normality  of  direct  labor 
efficiency  variances  experienced  at  a  medium  sized  manufacturing  plant.  Thirty-three  to 
42  weeks  of  direct  labor  efficiency  variances  were  collected  fi'om  each  of  14  production 
departments.  The  authors  did  not  describe  their  collection  method.  Thus,  their  sample  of 
variances  may  not  have  been  randomly  selected. 

Because  variance  thresholds  can  be  in  dollars  or  in  percentages,  the  authors  computed 
the  variances  both  ways.  A  direct  labor  efficiency  variance  expressed  in  dollars  is  the 
difference  between  the  planned  and  actual  number  of  hours  required,  multiplied  by  a 
standard  wage  rate.  The  direct  labor  efficiency  variance  can  then  be  converted  into  a 
percentage  by  dividing  it  by  the  actual  direct  labor  cost. 

Using  the  Bowman-Shenton  and  Shapiro-Wilk  normality  tests,  Gribbon  and  Lau  tested 
the  normalilty  of  the  variances  at  the  .05  significance  level,  and  had  mixed  results;  seven  of 
the  14  direct  labor  dollar  variances  were  normal,  and  only  1  of  the  14  direct  labor 
percentage  variances  were  normal. 

Assessment.  The  results  of  both  studies  indicate  that  cost  variances  are  not  always 
normal.  Neither  result  appears  to  be  based  on  a  random  sample  of  cost  variances,  and 
neither  result  was  based  on  defense  cost  variances.  Thus,  there  appears  to  be  ample  room 
for  this  study,  which  tests  the  normality  of  defense  cost  variances.  Indeed,  this  study  was 
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partially  prompted  by  the  advice  of  Gribbin  and  Lau  for  more  empirical  research  to 
validate  their  non-normality  conclusions  ( 1 3 :97). 

Conclusion 

This  chapter  has  reviewed  the  literature  pertaining  to  cost  variance  investigation 
models,  and  described  the  only  two  reported  empirical  tests  of  the  normality  assumption. 
The  statistical  models  show  considerable  promise  to  reduce  the  number  of  frivolous  cost 
variance  investigations.  However,  these  models  have  not  been  widely  adopted  in  industry, 
perhaps  because  the  information  requirement  is  quite  large  relative  to  the  information 
required  by  the  simple  model. 

Many  of  the  statistical  models  require  information  about  the  distribution  of  the  cost 
variance.  Often,  the  models  assume  that  the  distribution  is  normal.  Yet  the  only  two 
empirical  tests  of  this  assumption  show  that  cost  variances  are  sometimes  not  normally 
distributed.  The  following  chapter  will  describe  the  procedures  used  to  test  the  normality 
assumption  on  defense  cost  variances. 
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ffl.  Methodology 


Introduction 

This  study  tests  the  hypothesis  that  defense  cost  variances  are  normally  distributed. 
A  defense  cost  variance  is  defined  as  the  difference  between  the  Budgeted  Cost  of 
Work  Performed  (BCWP)  and  the  Actual  Cost  of  Work  Performed  (ACWP): 

Cost  variance  =  BCWP  -  ACWP  (1) 

ACWP  is  ‘hosts  actually  incurred  and  recorded  in  accomplishing  the  work  performed 
within  a  given  time  period.  ”(8:2-1).  BCWP  is  “the  sum  of  budgets  for  completed  work 
packages  and  completed  portions  of  open  work  packages”  and  coincides  to  the  same 
time  period  as  ACWP  (8:2-2). 

The  rationale  for  the  hypothesis  was  described  in  Chapter  I,  and  the  relevant 
literature  involving  the  statistical  cost  variance  investigation  models  and  prior  studies 
similar  to  this  one  were  reviewed  in  Chapter  H.  This  chapter  focuses  on  the  specific 
methodology  used  to  test  this  hypothesis  by  describing  the  statistical  normality  tests,  the 
sample  data,  and  the  procedures  used  to  collect  the  sample  data. 

Normality  Tests 

There  are  many  tests  of  normality.  In  a  comprehensive  review,  D’Agostino  and 
Stephens  concluded  that  no  single  test  is  optimal  for  every  possible  situation  (6).  Four 
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tests  were  used  in  this  study:  Bowman-Shenton  (simultaneously  uses  the  skewness  and 
kurtosis  moments),  Shapiro-Wilk,  Chi-square,  and  Kolmogorov-Smimov.  As 
described  in  Chapter  H,  Jacob  and  Lorek  evaluated  cost  variance  normality  using 
Inoment  tests, "which  involve  separate  measures  of  skewness  and  kurtosis,  and  the 
Kolmogorov-Smimov  test.  In  a  more  recent  study,  Gribbin  and  Lau  used  two  tests. 
The  first  was  the  Bowman-Shenton  test,  an  ‘bmnibus  moment  test”  which  combines 
skewness  and  kurtosis.  The  second  was  the  Shapiro-Wilk  W  test,  a  regression  test  of 
normality  recommended  by  D’Agostino  and  Stephens  (6).  Finally,  the  Chi-square  test 
was  used  largely  because  of  its  availability  in  statistical  software  packages. 

Skewness.  Skewness  is  a  measure  of  a  distribution’s  deviation  from  symmetry. 

The  normal  distribution  is  symmetrical,  with  the  mean,  median,  and  mode  the  same.  A 
distribution  that  stretches  toward  one  tail  or  the  other  is  termed  “Skewed.”  When  the 
tail  stretches  to  the  left,  toward  smaller  values,  it  is  negatively  skewed  where  the 
distribution’s  mean  <  median  <  mode.  When  the  tail  stretches  toward  the  right, 
toward  larger  values,  it  is  positively  skewed  where  mean  >  median  >  mode. 

The  equation  for  skewness  of  a  sample  is  (6:279,  375) : 

Vb,  =  [Z  (X;  -  tf]  /  [Z  (Xi  -  (2) 

where  bj  is  the  skewness  of  a  sample,  X;  is  a  random  variable  or  observation  for  i  =  1 
to  n,  and  u  is  the  sample  mean.  If  a  distribution  is  symmetric  about  its  mean,  as  is  the 
normal  distribution,  its  skewness  is  zero.  Thus,  a  non-zero  value  for  Vbi  indicates  that 
the  distribution  is  not  normal. 
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Kurtosis.  Kurtosis  is  a  measure  of  a  distribution’s  peakedness  (or  flatness). 
Distributions  where  dollar  variances  cluster  heavily  or  pile  up  in  the  center  (along  with 
more  observations  than  normal  in  the  extreme  tails)  are  peaked  or  ‘leptokurtic.”  Flat 
distributions  with  dollar  variances  more  evenly  distributed  and  tails  fatter  than  a  normal 
distribution  are  called  ‘J)latykurtic.’’  Intermediate  or  ‘hiesokurtic” distributions  are 
neither  too  peaked  nor  too  flat. 

The  equation  for  the  kurtosis  of  a  sample  is  (6:279,  375): 

b2  =  P(Xi-u)^/p(Xi-u)Y  (3) 

where  b2  is  the  kurtosis  of  a  sample,  Xj  is  a  random  variable  or  observation  for  i  =  1 
to  n,  and  u  is  the  sample  mean.  The  value  of  kurtosis  for  a  normal  distribution  is  3 
(6:375).  Values  of  62  not  equal  to  3  indicate  non-normality.  In  distributions  with  tails 
thicker  than  tails  in  the  normal  distribution,  b2  >  3.  Similarly,  when  the  tails  are 
thinner  than  tails  in  a  normal  distribution  b2<  3. 

Bowman-Shenton  Test.  The  Bowman-Shenton  test  consists  of  computing  skewness 
(Vbi)  and  kurtosis  (b2)  using  equations  (2)  and  (3),  and  plotting  the  couplet  (Vbi,  b2)  on 
a  contour  chart  drawn  for  a  given  level  of  significance.  D’Agostino  and  Stephens 
indicate  that  the  simple  moment  tests  for  normality  can  give  conflicting  signals  because 
skewness  (Vbi)  kurtosis  (b2)  are  not  independent  variables,  and  consider  the 
timnibus  test” developed  by  Bowman  and  Shenton  to  be  more  powerful  (6:283).  If  the 
plotted  point  is  external  to  the  contour  corresponding  to  the  sample  size,  the  null 
hypothesis  of  normality  is  rejected.  Both  90%  and  95%  contour  charts  are  provided  by 
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D’Agostino  and  Stephens  (6:282),  and  will  not  be  duplicated.  Here,  a  significance 
level  of  .05  ( a  =  .05)  was  selected  for  all  of  the  normality  tests,  and  the  95  %  contour 
chart  was  used  for  this  test. 

Shapiro-Wilk  Test.  The  Shapiro-Wilk  W  test  is  a  regression  test  of  normality.  For 
a  description  of  the  regression  procedures,  see  D’Agostino  and  Stephens  (6:393-394). 
The  W  test  statistic  is  computed  as 

W  =  (E  ai  Xi )'  /  2  (Xi  -  u)"  (4) 

where  a;  are  optimal  weights,  x,  is  the  random  variable  or  observation  for  i  =  1  to  n, 
and  u  is  the  sample  mean.  The  aj  values  were  derived  by  Shapiro  and  Wilks  using 
weighted  least  squares  regression  analysis,  and  are  available  in  tables  (e.g.,  6:209  and 
28:604). 

The  W  statistic  is  interpreted  similar  to  the  coefficient  of  determination,  R^.  The 
upper  limit  is  one,  and  the  closer  the  W  statistic  is  to  one,  the  closer  the  distribution  fits 
a  normal  distribution.  In  this  case,  the  larger  the  W  statistic,  the  closer  the  distribution 
of  cost  variances  is  to  normality. 

The  computed  W  test  statistic  is  compared  with  critical  W  values  in  a  table  provided 
by  several  authors  (4:468-469;  6:212;  28:605).  If  the  computed  W  test  statistic  is  less 
than  the  critical  value  given  in  the  lower  tail  of  the  table,  the  null  hypothesis  of 
normality  is  rejected.  For  example,  for  a  sample  size  of  48,  the  critical  value  is  0.947 
at  the  .05  level  of  significance.  If  the  W  test  statistic  is  less  than  0.947,  the  null 
hypothesis  is  rejected. 
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Kolmogorov-Smirnov  Test.  The  Kolmogorov-Smimov  test  for  goodness-of-fit 
(4:346-349;  5:650-651;  21:712-713)  compares  an  observed  sample  distribution, 

Fo(X),  with  a  theoretical  distribution,  Ft(X).  The  theoretical  distribution  represents  the 
expectation  of  normality  under  the  null  hypothesis.  The  test  determines  the  greatest 
vertical  distance  between  the  observed  and  theoretical  distributions,  and  defines  this 
value  as  maximum  deviation  (D). 

Using  a  table  of  critical  values  for  D,  the  test  determines  whether  such  a  large 
divergence  is  likely.  Conover  cautions  that  when  the  sample  size  is  larger  than  40,  the 
critical  value  is  not  exact,  but  can  be  approximated  by  a  formula  given  in  the  footnotes 
to  his  table  (4:462).  Here,  the  expected  sample  size  is  48.  Using  his  formula,  the 
critical  value  is  0. 192  at  a  significance  level  of  0.05.  Thus,  if  the  Kolmogorov- 
Smimov  test  statistic  exceeds  0.192,  the  null  hypothesis  of  normality  is  rejected. 

Chi-square  Goodness-of-Fit  Test.  The  chi-square  goodness-of-fit  test  compares  the 
observed  frequencies  (FJ  of  a  particular  occurrence  with  the  expected  frequencies  (Fg) 
of  the  assumed  distribution  to  determine  if  the  expected  distribution  fits  the  data.  The 
computation  for  the  test  statistic  (X^)  is  the  sum  of  the  observed  minus  expected 
frequencies  squared,  divided  by  the  expected  frequency  (5:447;  21:680): 

X^  =  S[(F.-F.)VfJ  (5) 

The  chi-square  statistic  is  based  on  the  size  of  the  difference  for  each  category  in 
the  frequency  distribution.  If  the  observed  frequencies  are  very  close  to  the  expected 
frequencies,  then  the  chi-square  statistic  will  be  close  to  zero.  As  the  observed 
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frequencies  reflect  greater  differences  from  the  expected  frequencies,  the  value  of  the 
chi-square  statistic  becomes  larger. 

The  level  of  significance  and  the  degrees  of  freedom  determine  the  critical  value  for 
the  chi-square  test  statistic.  The  degrees  of  freedom  are  equal  to  the  number  of 
categories,  minus  the  number  of  parameters  used  in  the  estimate,  minus  one.  The 
subtraction  of  one  is  necessary  because  the  last  category  entered  is  not  free  to  vary.  If 
the  test  statistic  exceeds  the  critical  value,  the  null  hypothesis  is  rejected. 

Software.  The  normality  tests  were  accomplished  with  a  micro-computer  and  three 
software  packages  available  at  AFIT:  Excel,  Statgraphics,  and  Statistix.  Excel  (27) 
was  used  to  compute  the  descriptive  statistics  (mean,  median,  mode,  skewness,  and 
kurtosis)  for  each  sample  of  cost  variances,  and  to  perform  the  Bowman-Shenton  test. 
Statgraphics  (29)  was  used  for  the  Kolmogorov-Smimov  and  the  Chi-square  tests. 
Statistix  (30)  was  used  for  the  Shapiro- Wilk  test. 

The  Data 

The  Cost  Performance  Report.  Data  for  the  normality  tests  were  obtained  from 
microfiche  copies  of  Cost  Performance  Reports  stored  in  the  cost  library  supporting  the 
Aeronautical  Systems  Center  (ASC)  of  Air  Force  Material  Command  located  at 
Wright-Patterson  Air  Force  Base.  Defense  contractors  prepare  the  Cost  Performance 
Report  (CPR)  each  month  and  send  it  to  the  system  program  office  that  manages  the 
project. 
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The  CPR  summarizes  the  cost,  schedule,  and  technical  status  of  the  defense  project 
using  a  standardized  breakdown  of  the  work  on  the  project,  termed  a  “Work  Breakdown 
Structure”  (WBS).  The  WBS  is  a  product-oriented  description  of  all  work  required  to 
complete  the  project,  and  is  often  viewed  as  a  family-tree,  with  successive  layers  of 
detail  termed  “levels”  (9). 

Since  1967,  CPRs  on  virtually  all  significant  defense  contracts  managed  by  ASC 
have  been  sent  to  the  program  offices  at  ASC  and  eventually  to  the  cost  library  for 
storage  on  microfiche.  The  CPR  typically  contains  monthly  and  cumulative  cost, 
budget,  and  variance  data  for  every  WBS  element  down  to  level  three,  although  the 
contractor  performs  work  at  much  more  detailed  levels. 

Validity.  To  ensure  the  validity  of  the  data  on  the  CPR,  the  DOD  requires  that  the 
contractor  comply  with  the  DOD  Cost/Schedule  Control  Systems  Criteria  (C/SCSC),  or 
“criteria”  for  short.  The  criteria  are  internal  controls  intended  to  ensure  that  the 
contractor’s  management  control  systems  provide  reliable  and  timely  data  useful  for 
managing  the  defense  contract  (2,  7,  8,  9,  14;  16:669-670).  Government  review  teams 
from  the  program  office  and  government  surveillance  teams  at  the  contractor’s  factory 
monitor  the  contractor’s  compliance  to  the  criteria.  If  the  contractor  is  compliant,  the 
government  assumes  that  the  data  on  the  CPR  are  reliable.  The  criteria  have  been 
required  since  1967,  and  most  defense  contractors  have  been  criteria-compliant  for 
many  years. 
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The  Collection  Procedure.  For  this  study,  about  4  years  of  monthly  cost  variances 
were  extracted  from  50  WBS  elements  on  two  research  and  development  contracts, 
termed  A  and  B.  The  identity  of  each  contract  will  not  be  revealed.  Forty-eight 
months  of  consecutive  cost  variances  were  considered  necessary  to  properly  replicate 
the  number  of  sequential  cost  variances  collected  by  Gribbin  and  Lau  (13).  Due  to 
severe  time  constraints  on  the  researcher,  only  two  contracts  with  the  necessary  4  years 
of  consecutive  cost  variances  were  selected.  Contract  A  contained  13  WBS  elements 
and  Contract  B  contained  37  WBS  elements  with  4  years  of  consecutive  cost  variances. 
Thus,  data  from  50  WBS  elements,  each  with  about  4  years  of  consecutive  cost 
variances,  were  manually  extracted  from  the  microfiche  and  input  into  an  Excel 
spreadsheet  for  analysis. 

Inflation  Adjustment.  Cost  data  on  CPRs  are  in  then-year  dollars.  It  was  not  clear 
if  the  cost  variances  needed  to  be  adjusted  to  constant  dollars.  Neither  of  the  two 
previous  normality  studies  (13,  17)  indicated  that  the  cost  variances  were  adjusted  for 
inflation  before  the  normality  tests  were  performed.  Further,  the  literature  describing 
the  cost  variance  investigation  models  does  not  address  this  issue. 

To  be  prudent,  the  normality  tests  were  performed  on  the  cost  variances  in  then- 
year  dollars  and  in  constant  dollars.  The  base  years  for  contracts  A  and  B  were  1991 
and  1974,  respectively.  Weighted  inflation  indices  corresponding  to  these  base  years 
were  available  from  an  internet  site  managed  by  the  Assistant  Secretary  of  the  Air 
Force,  Financial  Management  &  Comptoller  (SAF/FM)  in  Washington  D.C.  Once 
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down-loaded  and  entered  into  Excel,  the  cost  variance  data  were  converted  into  constant 
dollars. 

Conclusion 

This  chapter  has  described  the  procedures  for  testing  the  null  hypothesis  that 
defense  cost  variances  are  distributed  normally.  Based  largely  on  what  previous 
researchers  have  used  on  comparable  studies,  four  tests  were  selected  and  briefly 
described.  Among  these  are  the  Bowman-Shenton  and  the  Shapiro-Wilk  tests,  which 
are  considered  by  D’Agostino  and  Stephens  (6)  to  be  the  most  powerful  goodness-of-fit 
tests  for  normality.  The  only  two  known  normality  studies  reported  in  the  literature 
were  also  reviewed.  This  study  is  a  replication  of  the  most  recent,  performed  by 
Gribbin  and  Lau  (13).  Finally,  the  data,  the  data  collection  procedures,  and  the 
inflation  adjustment  procedures  were  described.  The  next  chapter  reports  the  results  of 
the  normality  tests. 
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IV.  Results 


Introduction 

This  chapter  describes  the  results  of  testing  the  null  hypothesis  that  cost  variances 
on  defense  contracts  are  distributed  normally.  Fifty  series  of  monthly  cost  variances 
experienced  on  two  defense  contracts  (Contract  A  and  Contract  B)  were  evaluated  using 
four  tests  of  normality  (Bowman-Shenton,  Shapiro-Wilk,  Kolmogorov-Smimov,  and 
Chi-square).  The  rationale  for  the  hypothesis  was  provided  in  Chapter  I.  The  relevant 
literature  was  reviewed  in  Chapter  H.  The  normality  tests,  data,  and  data  collection 
procedures  were  described  in  Chapter  HI.  Here,  several  tables  and  figures  are  used  to 
summarize  the  results. 

Four  tables  summarize  the  results  of  the  normality  tests.  Tables  3  and  4  pertain  to 
defense  contracts  A  and  B,  respectively,  with  the  cost  variances  reported  in  nominal 
dollars.  Tables  5  and  6  are  similar,  except  the  cost  variances  were  adjusted  to  constant 
dollars  before  applying  the  normality  tests. 

Each  table  is  formatted  the  same  way  to  facilitate  comparison  across  contracts.  The 
first  four  columns  list  the  work  breakdown  structure  (WBS)  element  number,  the  WBS 
level,  the  WBS  element  name,  and  the  final  Budget  at  Completion  (BAC)  of  that  WBS 
element  for  each  series  of  cost  variances.  The  next  four  colunms  contain  descriptive 
statistics  pertaining  to  the  cost  variances,  and  include  the  mean,  standard  deviation. 
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Table  6.  Results  of  Normality  Tests  on  Contract  B  (47  months,  Constant  Dollars) 
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median,  and  mode.  The  remaining  columns  contain  the  statistics  resulting  from  the 
four  normality  tests. 

In  addition  to  these  tables,  four  figures  pertaining  to  the  Bowman-Shenton 
normality  test  are  provided.  In  the  Bowman-Shenton  test,  measures  of  the 
distribution’s  shape  (skewness  and  kurtosis)  are  plotted  on  a  contour  chart  drawn  for  a 
specific  level  of  statistical  significance.  For  this  study,  the  level  of  significance  was 
five  percent  ( a  =  .05)  for  each  normality  test,  including  the  Bowman-Shenton  test. 
When  the  couplet  of  skewness  ( Vbi)  and  kurtosis  (b2)  lies  within  the  contour 
corresponding  to  the  sample  size,  the  distribution  of  cost  variances  is  normal. 

A  description  of  the  results  of  the  normabty  tests  follows,  first  for  Contract  A,  and 
then  for  Contract  B.  The  chapter  concludes  by  comparing  the  results  of  this  study  with 
results  reported  on  the  two  prior  studies. 


Contract  A 

As  shown  in  Tables  3  and  5,  the  null  hypothesis  was  generally  rejected  for  each  of 
the  thirteen  series  of  monthly  cost  variances  on  Contract  A  evaluated  in  nominal  and 
constant  dollars,  respectively.  Each  series  was  for  48  months  (January  1991  to 
December  1994).  Ten  of  the  series  were  at  WBS  level  3,  two  were  at  level  2,  and  the 
last  was  at  the  total  contract  level.  The  final  Budget  at  Completion  (BAC)  for  the 
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series  ranged  from  $127  thousand  (nominal  dollars)  for  WBS  Element  13  to  $184.8 
million  (nominal  dollars  )  for  WBS  Element  1 . 

In  general,  these  results  were  insensitive  to  the  normality  test  used  and  to  inflation. 
The  four  normality  tests  were  usually  in  agreement,  and  when  the  null  hypothesis  of 
normality  was  rejected  with  the  variances  in  nominal  dollars,  it  was  also  rejected  with 
the  variances  in  constant  dollars.  The  only  exception  was  WBS  Element  9  (System 
Software),  which  passed  the  Kolmogorov-Smimov  test  in  nominal  dollars  and  in 
constant  dollars  (but  failed  the  other  three  normality  tests). 

All  but  one  of  the  moment  couplets  for  the  Bowman-Shenton  test  were  off  the  95% 
contour  chart.  The  series  for  WTBS  Element  13  (Training)  was  on  the  chart  in  nominal 
and  in  constant  dollars,  but  because  it  was  outside  the  contour  line  corresponding  to  a 
sample  size  of  48,  the  series  was  not  normally  distributed  (Tables  3  and  5;  Figures  1 
and  3). 

None  of  the  Shapiro-Wilk  test  statistics  were  above  the  critical  value  of  0.947.  In 
general,  converting  a  series  from  nominal  to  constant  dollars  increased  the  Shapiro- 
Wilk  test  statistic,  moving  the  series  closer  to  normality,  but  never  enough  to  exceed 
the  critical  value  at  the  .05  level  of  significance. 

The  Chi-square  test  statistic  did  not  indicate  that  any  of  the  series  were  normally 
distributed.  However,  the  statistic  was  found  to  be  very  sensitive  to  the  software 
package  that  was  used.  When  computing  the  statistic  on  the  same  series  using  several 
statistical  software  packages,  different  Chi-square  statistics  were  reported.  An  analysis 
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Figure  1.  95%  Contour  Chart  for  Contract  A 
Then  Year  Dollars 

Source:  D’Agostino  and  Stephens  (1986,  p.  282),  with 
permission  of  the  authors  and  the  publisher,  Marcel  Dekker,  Inc. 
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Figure  2.  95%  Contour  Chart  for  Contract  B 
Then  Year  Dollars 

Source:  D’Agostino  and  Stephens  (1986,  p.  282),  with 
permission  of  the  authors  and  the  publisher,  Marcel  Dekker,  Inc. 
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showed  that  each  package  determined  a  different  number  of  cells  or  groups  in  which  to 
place  the  data.  Apparently,  each  package  used  a  different  and  undocumented  algorithm 
to  determine  the  number  of  cells.  For  consistency,  a  single  package  {Statgraphics  for 
DOS)  was  chosen  for  all  of  the  Chi-square  tests.  But  it  was  clear  from  this  analysis 
that  of  the  four  normality  tests  used  here,  the  Chi-square  test  is  the  least  appropriate. 

Contract  B 

As  shown  in  Tables  4  and  6,  the  null  hypothesis  was  generally  rejected  for  the  37 
series  of  monthly  cost  variances  on  Contract  B  evaluated  in  nominal  and  constant 
dollars,  respectively.  Each  series  was  47  months  (January  1977  to  December  1991, 
with  one  month  deleted  because  of  missing  data).  Seven  of  the  series  were  at  WBS 
level  4,  twenty-four  were  at  level  3,  five  were  at  level  2,  and  the  last  was  at  the  total 
contract  level.  The  final  BAC  for  the  series  ranged  from  $146  thousand  (nominal 
dollars)  for  WBS  Element  19  to  $152.2  million  (nominal  dollars  )  for  WBS  Element  1. 

As  with  Contract  A,  these  results  were  generally  insensitive  to  the  normality  test 
used  and  to  inflation.  But  there  were  some  exceptions.  For  example,  in  nominal 
dollars,  six  series  (WBS  elements  9,  13,  14,  15,  19  and  21 )  were  found  to  have 
normal  distributions  based  on  the  Bowman-Shenton  test,  and  non-normal  distributions 
based  on  the  Shapiro-Wilk  test.  As  shown  in  Table  4  and  Figure  2,  these  WBS 
elements  were  within  the  appropriate  contour  line  on  the  95%  contour  chart.  In 
constant  dollars,  four  series  (WBS  elements  9,  15,  16  and  23)  were  found  to  have 
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normal  distributions  based  on  the  Bowman-Shenton  test  ( Table  6  and  Figure  4),  but 
only  two  series  (WBS  elements  13  and  24)  were  normal  based  on  the  Shapiro- Wilk  test. 

The  Kolmogorov-Smimov  test  found  more  series  normal  than  the  other  tests.  In 
nominal  dollars,  ten  series  (WBS  elements  13,  19,  20,  24,  27,  30,  31,  33,  34,  37) 
tested  normal.  This  test  was  also  the  most  sensitive  to  inflation.  In  constant  dollars, 
six  series  (WBS  elements  19,  20,  27,  31,  33,  34)  were  normal. 

Like  the  Shapiro- Wilk  test,  the  Chi-square  test  found  few  series  to  be  normally 
distributed.  In  nominal  dollars,  only  one  series  (WBS  Element  20,  System  Test  and 
Evaluation)  was  normally  distributed.  In  constant  dollars,  only  one  series  (WBS 
Element  24,  Fatigue  Articles  Test)  was  normally  distributed. 

Comparison  to  Prior  Studies 

Table  7  compares  the  results  of  Jacobs  and  Lorek  (1980,  17)  and  Gribbin  and  Lau 
(1991,  13)  with  the  results  of  this  study.  Because  neither  of  the  prior  studies  referred 
to  any  adjustment  for  inflation,  it  is  assumed  that  no  adjustment  was  made.  To  be 
comparable,  the  results  of  this  study  are  presented  based  on  the  analysis  of  cost 
variances  in  nominal  dollars. 

Jacobs  and  Lorek.  The  results  reported  by  Jacobs  and  Lorek  are  the  most  different 
from  the  results  of  this  study.  For  Jacobs  and  Lorek,  78  percent  (7  of  9)  of  the  weekly 
series  had  normal  distributions  based  on  the  Kolmogorov-Smimov  test.  Here,  only  22 
percent  (1 1  of  50)  of  the  series  were  normal  using  the  same  test  (Table  3  and  4  for 
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nominal  dollars).  This  difference  may  be  due  to  differences  in  the  type  and  frequency 
of  the  variances  tested.  As  shown  in  Table  7,  the  variances  tested  by  Jacobs  and  Lorek 
were  not  in  dollars  and  were  not  monthly. 

In  addition  to  these  differences,  the  majority  of  the  defense  cost  variances  were 
much  more  aggregated  at  WBS  level  3  than  the  variances  tested  by  Jacobs  and  Lorek. 
As  defense  cost  variances  are  aggregated  from  levels  where  work  is  performed  (usually 
much  lower  than  WBS  level  4)  to  the  CPR  reporting  levels  (WBS  1,2,3,  and  4),  there 
may  be  some  loss  of  normality. 

Table  7.  A  Comparison  of  Cost  Variance  Normality  Studies 


Normality  tests  used 

Results  at  a  =  ,05 

Jacobs  and  Lorek 
(1980) 

Material  and  utilities  usage 
(11  daily,  9  weekly)  from 
a  grain  processing  firm 

Skewness,  Kurtosis, 
Kolmogorov-Smimov 

None  of  the  daily,  and  7 
of  9  weekly  series  tested 
normal. 

Gribbin  and  Lau 
(1991) 

Direct  labor  efficiency  in 
dollars  and  percent  (32  to 

43  months  in  each  of  14 
production  departments) 

Bowman-Shenton, 

Shapiro-Wilk 

7  of  14  dollar  series  and 

1  of  14  percentage  series 
tested  normal. 

Conley  (1996) 

Cost  variances  on  R&D 
defense  Contract  A  (48 
months  in  each  of  13  WBS 
elements) 

Bowman-Shenton 

Shapiro-Wilk 

Kolmogorov-Smimov 

Chi-square 

0  series  tested  normal 

0  series  tested  normal 

1  series  tested  normal 

0  series  tested  normal 

Cost  variances  on  R&D 
defense  Contract  B  (47 
months  in  each  of  37  WBS 
elements) 

Bowman-Shenton 

Shapiro-Wilk 

Kolmogorov-Smimov 

Chi-square 

6  series  tested  normal 

1  series  tested  normal 

10  series  tested  normal 

1  1  series  tested  normal 

This  is  apparent  even  at  the  reporting  levels.  For  example,  consider  the  Crew 
Station,  identified  in  Table  4  as  WBS  element  9.  Based  on  the  Bowman-Shenton  test, 
the  Crew  Station  was  the  only  level  4  element  to  be  normally  distributed.  The  other 
level  4  elements  were  not  normally  distributed.  The  parent  element  for  the  Crew 
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Station  and  the  other  level  4  elements  is  the  Air  Frame  at  level  3,  and  its  series  is  not 
normally  distributed.  Apparently,  as  many  non-normal  series  are  combined  with  few 
normal  series,  the  distribution  of  the  combined  series  may  not  be  normally  distributed. 

Gribbin  and  Lau.  With  the  possible  exception  of  the  level  of  aggregation,  the 
methodology  used  by  Gribbin  and  Lau  is  comparable.  Both  studies  examined  monthly 
cost  variances  in  dollars,  and  two  of  the  normality  tests  were  the  same.  In  addition,  the 
results  are  generally  consistent,  in  that  both  studies  found  a  significant  number  of  series 
to  be  non-normal.  As  shown  in  Table  7,  Gribbin  and  Lau  report  50  percent  (7  of  14) 
of  the  dollar  series  to  be  non-normal.  Here,  most  of  the  series  were  non-normal.  For 
example,  using  the  Shapiro-Wilk  test,  98  percent  (49  of  50)  of  the  series  were  non¬ 
normal,  and  using  the  Bowman-Shenton  test,  88  percent  (44  of  50)  of  the  series  were 
non-normal. 

As  before,  a  major  difference  between  Gribbin  and  Lau  and  this  study  pertains  to 
the  level  of  aggregation.  The  series  examined  by  Gribbin  and  Lau  are  direct  labor 
efficiency  variances.  The  series  examined  here  are  cost  variances,  defined  as  BCWP 
minus  ACWP  (Equation  1,  Chapter  El).  All  costs  may  be  included  in  these  numbers, 
including  direct  labor,  direct  material,  and  indirect  costs.  In  general,  the  Cost 
Performance  Report  will  not  distinguish  between  such  categories  at  WBS  levels  3  or  4. 
Such  detail  would  only  be  available  at  much  lower  levels  in  the  WBS,  and  is  generally 
not  provided  to  the  government  unless  specifically  requested.  Thus,  cost  variances 
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which  may  be  normally  distributed  at  the  more  detailed  levels  in  the  WBS  may  lose  this 
characteristic  as  they  are  aggregated  and  eventually  iqwrted  on  the  CPR. 

These  results  appear  to  conflict  with  the  Central  Limit  Theorem  in  statistics,  which 
infers  that  as  cost  variances  are  aggregated  from  lower-level  WBS  elements  into  higher- 
levels  in  the  WBS,  the  total  should  become  increasingly  normal.  Clearly,  that  is  not 
the  case  here. 


One  possible  explanation  is  a  lack  of  independence  among  the  lower  level  elements. 


To  test  this  possibility,  the  cost  variances  in  WBS  elements  within  the  Air  Frame  on 


Contract  B  were  tested  for  correlation  using  the  nonparametric  Spearman  rank 
correlation  test  (5:505-509;  29).  (The  alternative  parametric  Pearson  Product  Moment 
Correlation  test  (5:481-488;  29)  was  not  used  because  most  of  the  cost  variances  at 
WBS  level  4  were  not  normal.).  The  results  of  this  test  are  presented  in  Table  8. 


Table  8.  Spearman  Rank  Correlations  Between  WBS  Level  4  Elements 


WBS  element 

4 

5 

6 

7 

8 

9 

10 

4 

Integration  and 
Assembly 

1 

5 

Basic 

Structure 

0.2796 

0.0579 

1 

6 

Vehicle  Power 

-0.1179 

0.4241 

-0.0984 

0.5047 

1 

7 

Environmental 
Control  System 

0.2831 

0.0549 

-0.0326 

0.8249 

1 

8 

Flight  Control 
System 

-0.3486 

0.0181* 

0.4112 

0.0053* 

1 

m 

Crew  Station 

0.4583 

0.0019* 

0.0696 

0.6370 

-0.1513 

0.3049 

0.4333 

0.0033* 

BBI 

1 

0.0057 

0.9690 

-0.2412 

0.1019 

0.2446 

0.0971 

0.4334 

0.0033* 

1 

39 


The  first  number  in  each  row  of  the  table  is  the  Spearman  rank  correlation  coefiBcient 
for  the  WBS  elements  within  the  Air  Frame.  The  correlation  coefiBcients  range  between 
-1  and  +1,  and  measure  the  association  between  the  WBS  elements.  The  second  number 
in  each  row  is  the  statistical  significance  (p-value)  of  the  estimated  correlations.  Here,  a 
p-value  below  0.05  indicates  significant  non-zero  correlation,  and  the  null  hypothesis  of 
independence  is  rejected. 

As  shown  in  the  table,  several  of  the  WBS  elements  at  level  4  are  significantly 
correlated.  Most  of  the  significant  positive  correlations  seem  plausible;  the  one  negative 
correlation  may  not  be  plausible.  For  example,  the  Crew  Station  is  significantly  correlated 
with  the  Environmental  Control  System.  It  seems  reasonable  that  cost  variances  involving 
the  Crew  Station  could  be  dependent  on  cost  variances  involving  the  Environmental 
Control  System  because  the  WBS  elements  are  functionally  related. 

The  significant  negative  correlation  between  the  Flight  Control  System  and  the  Basic 
Structure  is  less  plausible,  but  explaining  the  relationship  is  not  the  purpose  of  this 
analysis.  Here,  the  purpose  is  to  discover  a  lack  of  independence  among  the  level  4  cost 
variances  within  the  Air  Frame  at  WBS  level  3.  The  lack  of  independence  may  explain 
why  the  Central  Limit  Theorem  does  hold  in  this  case.  Specifically,  as  cost  variances  are 
aggregated  up  the  WBS,  they  do  not  become  normal  because  at  least  some  of  the  WBS 
elements  are  not  independent. 


40 


Conclusion 


This  chapter  has  described  the  results  of  testing  the  null  hypothesis  that  cost 
variances  on  defense  contracts  are  distributed  normally.  With  few  exceptions,  the  null 
hypothesis  of  normality  was  rejected  at  the  .05  level  of  significance.  This  result  was 
generally  insensitive  to  the  normality  test  used  and  to  inflation  adjustments. 

Prior  studies  have  also  shown  that  cost  variances  are  not  always  distributed 
normally.  Here,  the  percentage  of  cost  variances  found  non-normal  was  significantly 
larger  than  percentages  reported  by  others.  This  difference  may  be  due  to  the  level  of 
aggregation  and  the  lack  of  independence  among  lower-level  WBS  elements.  The  cost 
variances  tested  here  were  at  a  much  higher  level  of  aggregation  (WBS  levels  4  and 
above)  than  the  variances  tested  previously. 

The  next  chapter  will  summarize  this  study,  discuss  its  implications,  and  propose 
areas  for  further  research. 
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V.  Conclusion 


Introduction 

This  study  was  prompted  by  increasing  defense  requirements  to  reduce  costs  and  find 
more  efficient  ways  of  doing  business.  Over  the  years,  there  have  been  many  DOD 
initiatives  to  economize,  including  the  application  of  DOD  Cost/Schedule  Control 
Systems  Criteria  or  “the  criteria”  on  major  defense  contracts.  Recently,  however,  the 
criteria  have  been  criticized  as  being  a  non-value  added  cost  to  defense  contracts.  In 
particular,  one  of  the  requirements  under  the  criteria,  cost  variance  analysis,  has  been 
criticized  as  an  administrative  burden  to  defense  contractors  which  results  in  fiivolous  and 
costly  variance  investigations  and  reports. 

Although  cost  variance  analysis  is  a  widely  used  management  control  practice,  it  can  be 
over-implemented.  In  theory,  cost  variances  should  be  investigated  only  when  benefits 
from  identifying  and  correcting  the  variance  exceed  the  cost  of  the  investigation.  To  this 
end,  a  number  of  statistical  cost  variance  investigation  models  have  been  described  in  the 
literature.  In  practice,  the  statistical  models  are  rarely  used  on  defense  contracts  or 
elsewhere.  Instead,  a  simple  investigation  model  is  used  where  a  variance  is  investigated 
when  it  breeches  a  pre-specified  threshold.  Although  the  simple  model  is  easy  to  use,  it 
can  prompt  a  fiivolous  investigation. 

One  reason  suggested  in  the  literature  for  not  using  the  statistical  models  is  that  the 
models  require  too  much  advance  information  about  the  cost  variance,  including  its 
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distributional  form.  The  literature  describing  these  models  commonly  assumes  that  the 
cost  variance  is  distributed  normally,  for  example.  But  the  models  may  not  work  properly 
if  this  assumption  is  fallacious: 

Gribbin  has  shown  recently  that  if  the  cost  variances  are  indeed  non-normal,  then 
assuming  normality  instead  of  modeling  the  non-normality  correctly  can  lead  to 
significantly  inferior  cost  variance  investigation  decisions.  (Gribbin  and  Lau, 
13:88) 

Given  the  criticism  over  the  present  cost  variance  model  used  on  defense  contracts,  and 
the  potential  benefit  from  adopting  a  statistical  cost  variance  investigation  model,  this 
study  tested  the  null  hypothesis  that  defense  cost  variances  are  distributed  normally.  If  the 
hypothesis  is  accepted,  then  using  the  statistical  models  described  in  the  literature  should 
be  encouraged.  If  the  hypothesis  is  not  accepted,  then  the  models  may  still  be  used,  but 
only  with  additional  information  about  the  distributional  form  of  defense  cost  variances. 

Chapter  n  described  the  statistical  cost  variance  investigation  models  which  have  been 
proposed  in  the  academic  literature.  In  addition,  two  prior  studies  which  have  tested  the 
normality  assumption  were  reviewed.  Neither  of  these  studies  focused  on  defense  cost 
variances,  and  each  had  mixed  results.  Some  series  of  cost  variances  were  normal;  others 
were  not.  This  study  replicated  these  studies  on  defense  contracts. 

Chapter  HI  described  the  methodology  related  to  testing  the  hypothesis.  Fifty  series  of 
cost  variances  from  two  defense  contracts  were  collected  and  tested  for  normality.  The 
sensitivity  of  the  results  to  the  specific  normality  test  used  and  to  the  effects  of  inflation 
were  analyzed. 
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As  described  in  Chapter  IV,  the  results  show  that  the  vast  majority  of  defense  cost 
variances  tested  were  not  distributed  normally.  In  general,  this  was  true  regardless  of  the 
normality  test  used  and  whether  or  not  the  variances  were  adjusted  for  inflation.  The 
implication  of  this  finding  and  suggestions  for  further  research  will  now  be  described. 

Implications  and  Further  Research 

The  results  show  that  most  of  the  cost  variances  on  the  Cost  Performance  Report 
(CPR)  for  two  defense  contracts  were  not  normally  distributed.  This  implies  that 
government  program  oflBces  cannot  safely  assume  normality  when  using  a  statistical  cost 
variance  investigation  model.  To  do  so  may  result  in  a  signal  to  investigate  when  an 
investigation  would  not  be  beneficial,  or  in  a  signal  to  not  investigate  when  an 
investigation  would  be  beneficial. 

A  non-normal  distribution  may  be  more  appropriate  for  statistical  cost  variance 
investigation  models  used  at  the  government  program  offices.  In  many  cases,  the 
literature  describing  these  models  assumes  normality  because  it’s  convenient.  Most  of  the 
models  could  use  non-normal  distributions. 

The  wording  of  this  conclusion  is  careful  and  deliberate.  Limitations  associated  with 
the  conclusion  affect  its  generahzability.  One  pertains  to  the  level  aggregation;  the  other 
pertains  to  random  sampling.  The  limitations  also  suggest  areas  for  further  research. 

Level  of  Aggregation.  The  results  of  prior  studies  were  mixed,  with  some  series  of 
cost  variances  normal,  and  other  series  not  normal.  Here,  almost  all  of  the  series  were  not 
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normal.  The  major  difference  between  this  study  and  the  others  pertains  to  the  level  of 
aggregation  of  the  cost  variances.  In  the  prior  studies,  the  variances  were  measured  where 
the  work  is  accomplished.  Cost  variances  at  the  working  level  typically  include  direct 
costs  only,  and  can  often  be  separated  into  material  and  labor  components. 

In  this  study,  the  cost  variances  were  measured  at  the  reporting  level,  which 
corresponds  to  levels  1  through  4  on  the  Work  Breakdown  Structure  (WBS)  used  to 
define  and  organize  work  on  a  defense  contract.  At  these  summary  reporting  levels,  the 
variances  can  include  hundreds  of  lower-level  WBS  elements,  and  often  combine  direct 
materials,  direct  labor,  and  various  kinds  of  indirect  costs. 

It  seems  reasonable  that  as  cost  variances  are  aggregated  up  the  WBS  from  the 
working  level  to  the  reporting  level,  a  normally  distributed  cost  variance  at  the  working 
level  could  loose  this  characteristic,  expecially  if  the  WBS  elements  at  the  working  level 
are  not  independent.  In  fact,  this  was  found  to  be  the  case  as  the  defense  cost  variances 
were  aggregated  from  WBS  level  4  to  higher  levels  on  the  CPR.  Thus,  the  conclusion  that 
defense  cost  variances  on  the  CPR  are  not  be  normal  does  not  extend  to  lower  levels  in 
the  WBS.  The  normality  of  defense  cost  variances  at  the  working  level  is  an  empirical 
question. 

Nonrandom  Sampling.  Neither  the  prior  studies  nor  this  one  employed  statistical 
random  sampling  to  identify  the  cost  variance  series  for  testing.  On  this  study,  a  random 
sample  was  not  practical  given  the  time  constraints  and  the  tedious  task  of  retrieving  the 
cost  data  from  microfiche.  It  was  simply  not  feasible  to  select  a  random  sample  of 
contracts  with  an  adequate  number  of  monthly  cost  variances.  Thus,  it  cannot  be  inferred 
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from  these  results  that  cost  variances  on  other  defense  contracts  are  non-normal.  This 
too,  remains  an  empirical  question. 

Conclusion 

The  results  show  that  defense  cost  variances  reported  on  the  Cost  Performance  Report 
are  not  always  normally  distributed.  Using  statistical  cost  variance  investigation  models  to 
signal  variance  analysis  is  still  feasible,  but  without  the  assumption  that  the  cost  variances 
are  distributed  normally.  Other  non-normal  distributions  should  be  explored.  Thus,  this 
study  may  be  extended  to  identify  alternative,  non-normal  distributions  which  more  closely 
fit  cost  variances  on  CPRs.  Once  identified,  a  demonstration  of  the  statistical  model  which 
uses  the  non-normal  distribution  would  be  useful  to  those  contemplating  the  application  of 
the  model  in  a  government  program  office.  Additionally,  the  distributional  properties  of 
defense  cost  variances  below  the  reporting  level  is  not  known.  Once  known,  the  statistical 
models  again  promise  the  potential  to  reduce  the  number  of  frivolous  investigations 
conducted  by  the  defense  contractor. 
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Appendix:  Cost  Variance  Data 


This  Appendix  contains  cost  variance  data  extracted  from  monthly  Cost  Performance 
Reports  on  two  defense  contracts,  identified  as  Contract  A  and  Contract  B.  The  identity 
of  the  contracts  is  not  revealed  to  ensure  anonymity.  Tables  9  and  10  each  contain  the 
Work  Breakdown  Structure  (WBS)  element  number,  the  monthly  date  of  the  Cost 
Performance  Report,  the  weighted  inflation  index,  the  monthly  cost  variance  (C  V),  and 
the  Budget  at  Completion  (BAC),  for  Contracts  A  and  B,  respectively.  Because  the  WBS 
names  and  levels  corresponding  to  the  WBS  element  numbers  are  provided  in  Tables  3 
through  6  in  Chapter  IV,  they  are  not  repeated  here. 
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Table  9.  Contract  A  (Nominal  SOOO) 


Table  9.  Contract  A  (Nominal  $000)  -  continued  - 


Table  10.  Contract  B  (Nominal  $000) 


50 


Table  10.  Contract  B  (Nominal  $000)  -  continued 
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Table  10.  Contract  B  (Nominal  $000)  -  continued  - 


Table  10.  Contract  B  (Nominal  SOOO)  -  continued  - 


Table  10.  Contract  B  (Nominal  SOOO)  -  continued  - 
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